Variable-Based Calibration for Machine Learning Classifiers

The deployment of machine learning classifiers in high-stakes domains requires well-calibrated confidence scores for model predictions. In this paper we introduce the notion variable-based calibration to characterize properties a with respect variable interest, generalizing traditional score-based metrics such as expected error (ECE). particular, find that models near-perfect ECE can exhibit significant miscalibration function features data. We demonstrate phenomenon both theoretically and practice on multiple well-known datasets, show it persist after application existing methods. To mitigate issue, propose strategies detection, visualization, quantification error. then examine limitations current methods explore potential modifications. Finally, discuss implications these findings, emphasizing an understanding beyond simple aggregate measures is crucial endeavors fairness interpretability.